Field
The present invention relates to the field of automated computerized driver assistance for vehicles. The invention regards in particular a method and a corresponding program for computationally performing scene analysis of semantic traffic spaces based on an adaptive spatio-temporal ray based approach, the vehicle equipped with such a system and the system itself.
The invention is in the field of driver assistance based on for example at least one of computer vision, pattern recognition, classification and machine learning. The invention can in particular be implemented in a sensor-based computing module, which can be part of a car, a motorbike, or any other vehicle. The invention can be applied in realistic real-world traffic environments, such as encountered when driving a car in an unconstrained inner-city scenario.
An automated driver assistance system in a vehicle in most cases includes a sensor physically sensing the environment of the vehicle, and a computing unit, supplied with a sensor output signal, and computing an output signal which assists the driver in steering and controlling of the vehicle. The output signal may be supplied to optical or acoustical indication means and/or to a controller unit for controlling an actuator of the vehicle. An actuator of the vehicle might be a part of safety device, for example an inflator for an airbag system, or an actuator influencing the movement of the vehicle, for example brakes, accelerator, steering.
Automated driver assistance systems (ADAS) such as an “Adaptive Cruise Control” (ACC) system (for example described in ISO-Norm 15622:2010) increase driver comfort and safety. ACC systems are especially used for carrying out longitudinal control of a host vehicle, for example with respect to a target speed specified by the driver and ranging to other traffic objects such as other vehicles, for example cars, motorbikes, bikes, trucks or pedestrians. Driver assistance systems, for example lane change assistants, are based on predicting a future behavior of other participants in a traffic environment of a vehicle with aid of a prediction system.
The term “ego vehicle” will be used in the following description for a vehicle in a traffic situation which has a prediction system comprising the invention mounted thereon and which is equipped with the sensors and/or further means for acquiring traffic related data and a computing system that allows the computation of a possible or likely future behavior of at least one other traffic vehicle. The ego-vehicle is sometimes also referenced as host vehicle.
A sensor may be any means that can deliver information suitable for describing a traffic scene at a point in time by physically sensing an environment of the host vehicle. Such sensors may include one or more cameras, radar, lidar, laser scanner or the like. A data acquisition means may form part of the driver assistance system for acquiring any kind of information describing the traffic scene provided by one or more sensors or other data sources. In modern vehicles a human driver is often assisted by “active safety systems”. Such active safety systems (in the following also referred to as “driver assistance systems”) which can be a lane keeping assistance system physically sense the environment of the vehicle and extract information that is necessary for performing the driver assistance function. Based on this sensor signal processing the driver assistance system outputs a signal which can be fed to visual and/acoustic representations means, or it can be fed to an actuator (steering, brakes, safety belt pre-tensioning, airbag, . . . ) the action of which alters the state of the vehicle or its driving condition.
One type of such information on the environment of the vehicle that is highly relevant for driver assistance systems is the “road terrain”. The road terrain in the context of the present invention is understood as the type of surface of the road on which the vehicle drives or can potentially drive. The road terrain is thus a part of the environment that is semantically important for the task of driver assistance. The road terrain includes—on an upper level—also sidewalks, off-road terrain, traffic islands etc. All areas with surfaces accessible to a vehicle may form part of the traffic space available to a vehicle.
A further type of information about the environment of the vehicle that is important for driver assistance systems are surrounding objects. Such surrounding objects which a vehicle in a typical road scenario encounters are often elevated so that they cannot be driven over. Such elevated objects are e.g. other vehicles, buildings, trees and traffic signs.
Identifying a specific road terrain category by the above denoted types of information can in general be performed with a variety of sensors providing, for example, camera images, depth information or GPS/map data.
An evaluation of a vehicle's maneuvers or future maneuver options uses a representation of the vehicle's environment. One such representation of a vehicle's environment is in the form of an “occupancy grid”. An occupancy grid is map like description of the environment which is two-dimensional and in which each grid point constituting the occupancy grid includes information if it is occupied by an obstacle and if it belongs to road terrain. The occupancy grid (grid) may also be described by the plurality of individual grid cells constituting the grid.
Description of the Related Art
The detection of road terrain from acquired camera images by classifying selected locations is shown in EP 2 574 958 A1. Locations in a top-view projection of pixel confidences are selected as base points. A spatial feature generation of the base points is based on a value continuous confidence representation capturing visual and physical properties of the environment. Directed straight rays extend from the base points in different directions according to a fixed set of angular orientations. The extracted features are used for classifying road terrain using both local properties of sensor data and the spatial relationship in the feature extraction process. The classification results in assigning a semantic label such as “road terrain”, “own lane” to a part of the traffic space that can potentially be driven over.
In EP 0911779 A2 a general traffic situation is classified into categories (classes) such as “accident”, “congestion/traffic jam”, based on image processing of data acquired from traffic surveillance data.
DE 10 2012 008 659 A1 shows a situation analysis by classification of traffic situations from the perspective of other entities determining based on relative distances to other entities and to road boundaries and occupied spaces if the other entities will execute a cut-in or cut-out maneuver. DE 10 2012 008 659 A1 classifies based on acquired image data, data received via vehicle-to-vehicle and vehicle-to-infrastructure communication and uses an occupancy grid as well as map information. The classified traffic situations are used for predicting future behavior of the other entities.
The state of the art approaches for classifying a traffic scene take the perceived context and therefore acquired spatial data into consideration. The classification is therefore based only on such information which is directly measurable, for example in case of an optical sensor means which is visible. The classification according to the state of the art lacks therefore awareness with respect to features which are not directly measurable but nevertheless strongly influence the understanding of a traffic scene under analysis.
The known situation recognition ignores the overall spatial layout of a scene and only evaluates the relations between different entities in a traffic scene, possibly also representing the road as one of entities. Current scene classification methods either use conventional image processing methods in order to label a whole traffic scene or to relate traffic participants to each other to determine specific situations in a scene. However a spatial layout of the semantics of the sensed environment is disregarded, for example where is road is neglected, instead the prior art focus on explicit relations between entities, for example “is on road”.
While the sensor means are suitable to acquire spatial data which enable to perceive context of a traffic scene, for example lane markings, curbstones, road surfaces, the interpretation of the sensed spatial data is neglected in the prior art approach. The prior art relies on a perceived context for analysis of the traffic space. Further information referring to a semantic context is neglected. Semantic context interprets a perceived context in order to assign a category to elements in a traffic scene. In case of a perceived context identifying a “road”, the semantic context identifies for example the category “own lane area”.